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ABSTRACT 


The Space Shuttle will be equipped with a pair of 50 ft. manipula- 
tors used to handle payloads and to perform mechanical assembly operations. 
While current plans call for these manipulators to be operated by a 
human teleoperator, this article examines the possibility of using 
results from robotics and machine intelligence to automate this Shuttle 
assembly system. The major components of an autonomous mechanical 
assembly system are examined, along with the technology base upon which 
they depend. The state of the art in advanced automation is assessed 
in the Appendix. 
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I. INTRODUCTION: THE SPACE SHUTTLE - A WORKSHOP IN SPACE 


With the introduction of its Space Shuttle program NASA has 
initiated a new era in the utilization of space and space resources. 

Unlike all previous spacecraft, the Shuttle will act as a seed from 
which can blossom an entirely new class of challenging applications. 

Here we focus on a broad sub-class of these applications that involves 
handling and mechanical assembly, Construction of platforms, antennae, 
space stations, solar collectors, and other large structures in Earth 
orbit are examples. Each of these projects will require techniques 
for handling raw and fabricated materials, placing parts in precise 
translational and orientational relationships, and making the relation- 
ships permanent by fastening them in some way. In this article I examine 
the potential for application of techniques from advanced automation, 
robotics, and machine intelligence in making these assembly operations 
more automatic and less dependent on human intervention. As a result 
of employing these ideas, man's activities in space will become more 
productive, less expensive, and safer. 

There are already plans to equip the Space Shuttle with first 
one, and then a pair of general purpose manipulators that can be used 
in solutions to these handling and. assembly problems [33], Working 
in a special cockpit, "payload specialists" will control the 50-ft. 
manipulators from a pair of joy-sticks and a number of other raanipulanda, 
while observing progress through windows or closed circuit video. Other 
sensors mounted on the manipulators are being planned so that information 
can be displayed to the operator for interpretation and action [7]. 

These existing plans are for a teleoperator system that features 
man's active role in the feedback control loop. What we are investigating, 
however, is the replacement of man by automated computational machinery 
for many of the control, sensing, and sequencing operations required 
during these same assembly and handling operations. Applying techniques 
developed in advanced automation, robotics, and machine intelligence, 
and taking advantage of the computer's facility to deal with complicated 
sensing and control problems, it is possible to elevate man's role 
to that of an overseer who needs only to direct activities at an advanced 
level by issuing high order commands. 

The basic- idea here is to remove man from the role of a real-time 
sensory and control feedback element. This is done by replacing him 
with computational machinery, both hardware and software, that perform 
the required perceptual and control functions (see Fig. 1). In this 
way the human and computer each do what they do best: the human plans 

and makes high level decisions; the computer handles the myriad of 
details required by control of high order mechanical systems, optimizes 
energy consumption, ensures collision free trajectories, etc. While 
humans can be trained to accomplish many of tliese lower level tasks, 
we expect the fully automated manipulation system to out-perform the 
human teleoperator with respect to speed, dexterity, safety, and cost. 

Though the distinction between a teleoperator and a robotic 
system need not be subtle, many features of the conceptual approach 
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and hardware technologies found in these two disciplines are quite 
similar. It is not surprising, therefore, that one might find techniques 
from robotics combined with those from teleoperation in a single imple- 
mentation. Such compromises may eventually prove useful, but the thesis 
pursued here is that under normal circumstances man should make no 
real-time control decisions for a manipulator based assembly system. 
Furthermore, we encourage the use of automated techniques for supervision 
and sequence execution, since they often involve rather routine bookkeeping 
and coordination procedures. 

It is the purpose of this document to describe the major components 
of an automated mechanical assembly system, and to introduce general 
v"* requirements for each component. It is folly to attempt specification 
of requirements without first obtaining a clear idea of what the system 
is to do. Since detailed plans do not yet exist for space assemblies, 
three fictional, but realistic, examples are presented in the next section 
to guide the discussion. Once the major functional components are intro- 
duced, we examine, t-he specific technical elements that are necessary for 
successful assero’/iy. 

II. TASKS 

Current plans call for the Shuttle manipulators to be used 
for a range of handling and assembly tasks. Payloads will be retrieved 
from storage and placed in orbit or retrieved from orbit and stored. 
Carefully pre-packaged devices will be deployed, perhaps by operating 
mechanisms once the device has been safely placed clear of the Shuttle 
bay. Still other tasks require a manipulator to perform a sequence 
of assembly operations that produce a needed structural component 
from a pre-packaged kit. Each task requires different resources. 

We can describe the Shuttle automated assembly system as a 
pair of manipulators, a set of sensors, computational elements, a set 
of programmed algorithms, and perhaps a few special tools. All actions 
upon the environment are performed by the manipulators, and informative 
signals from the task environment are obtained from the various sensing 
elements. The manipulators are moved about so that raw materials and 
parts are transported, brought together, stacked, mated, fastened, 
and stored or unstored until the desired result is achieved. 

In the discussion that follows it is assumed that the logical 
design of each task, (i.e., the sequence of events and type of operation 
performed during each event) , is planned by human designers working 
on Earth during mission planning. Though looping and branchivig on 
sensed data are permitted, the major steps in an execution will proceed 
in a largely predetermined manner. 


THREE REPRESENTATIVE EXAMPLE TASKS 

In order to proceed with a concrete analysis, I introduce three 
example tasks that are representative of those to be performed using 
the Shuttle manipulators. The specific details of these tasks are 
not taken from actual plans, but are contrived to include a range of 
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generic features that such plana are expected to include. These examples 
serve as the focus for later discussion below: ^ 

TASK (A) Transfer a small, light satellite equipped with a grapple 
fixture from an orbit near the Space Shuttle to a storage place in 
the Shuttle bay. The initial crbit may include low velocity rotational 
and translational motion of the satellite relative to the Shuttle. 

These motions must be < ,5 rad/sec and < 1 m/sec respectively [28]. 

This operation requires execution of the following sequence of steps: 

1. Determine the motions of the payload relative to a Shuttle 
attached coordinate system. 

2. The manipulator moves^r^om a storage position in the Shuttle 
bay to the payload so that the end effector is in proper position 
relative to the payload. 'The accuracy of this positioning operation 
must be suph that subsequent activation of the grasp mechanism 

does not produce undue movement of the payload. A sensor must 
provide information regarding the precise position and velocity 
of the grapple fixture on the payload. 

3. The grapple fixture is activated. 

The manipulator continues to track motion of the payload 
while an acceleration force is applied to bring the object to 
rest rotational and translational velocities must be nulled. 

5. The manipulator and payload are then moved along a trajectory 
that avoids collision with objects in the bay, notably other 
stored items, to a vacant storage location. 

6. The grapple mechanism Is activated to release the payload. 

This must be done in a way that does not impart undesirable motion 
to the payload. It may also be necessary to fasten the payload 

to the bay in some way. 

7. The manipulator is returned to a resting position. 

Note that this scenario could be reversed in order to place a stored 
object into orbit. The requirements for this reverse process will 
probably not be the same, however. 

TASK (B) Move a subassembly from the bay into position so that it can be 
attached to a larger assembly under construction outside of the Shuttle 
(see Fig. 2). Completion of this task requires the following actions: 

1 . The manipulator moves from its resting position to a suitable 
grasp point. 
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Figure 2. Hating and Fastening a Subassembly to a 
Larger Structure Under Construction 
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2 . The gripper operates to grasp the subassembly. 

3. The manipulator control algorithms are adjusted for the additional 
mass and moment connected to the gripper link. 

A safe trajectory is chosen that moves the subassembly close to the 
goal position. Note that collisions of the subassembly with portions of the 
manipulator and with other obstacles in the environment must be avoided. 

5. The subassembly is moved to mate with the main assembly at 
the three connection points simultaneously. During this operation 
it is essential to maintain precise stability of the Shuttle 
relative to the main assembly, or to measure relative movements 
precisely. Highly accurate manipulator control is also essential. 

6. The second manipulator is used to fasten the subassembly 

to the main assembly at each point. This procedure may require 
a subsequence of operations, e.g.t 

Move AHM 2 from rest position to storage location of fastening tool. 

Move tool to first connestion point. 

. Fasten . 

Move tool to next connection point. 


Deposit tool in storage . 

Move ARM 2 to rest position. 

Naturally, this operation v;ould be much more complicated if extra 
parts were needed for fastening, or if ARM 2 could not reach 
all fasten points. 

TASK (C) Select beam-like parts, tension support wires, and connectors 
from a specially prepared kit, and assemble them into the subelements 
of a larger structure. (See Fig. 2.) This activity takes place in the 
bay where parts are 'stored. Fixtures are available when necessary. (In 
what follows it is assumed that obstacle free trajedtories are chosen.) 

1 . Determine precise location of first member . 

2. Hove manipulator from rest position to near grasp point of 
MEMBER 1 and grasp. 

3. Place MEMBER 1 in vise fixture. (Connectors are pre-mounted 
on ends of MEMBER 1 . ) 

4. Locate MEMBER 2 precisely and "grasp. 

5. Move MEMBER 2 into position and mate with MSM0£R 1 , This 
may involve alignment and insertion procedures. 

6. Fasten MEMBER 1 and MEMBER 2 using connector. At this point 
fixturlng is needed to provide structural suppprt for the unfinished 
subassembly since sufficient structural elements are not yet in place. 
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Figure 3> Example of Subassembly that is 
Constructed in the Shuttle Bay 
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7. Inspect progress and check for errors, 

8. Fetch and posi/iion MEMBER 3. 

9. Faaten MEMBER 3 to MEMBER 1 and MEMBER 2. Because MEMBER 3 
will be mated at both ends, movement of MEMBERS 1 and 2 may be 
required. This will depend on the exact nature of the connector 
mechanism. 

10. Inspect structure for correct shape and foi strength. 

11. Fetch, position, and attach MEMBER 

12.-18. Locate, grasp, and install Members H, 5, 6, 7» 8( and 9. 

Perform an inspection after each operation. 

.19. Attach one end of tension support CABLE A to connector 
at junction of MEMBERS 1, 2, and 11. 

20. Attach the other end of CABLE A to connector at intersection 
of MEMBERS 5, 8, and 9. The cable must be stretched to prescribed 
tension before fastening. An Iterative procedure that repeatedly 
tightens each cable may be required. This stretch-measure tension- 
fasten operation should probably be done with a special tool 
that is positioned by the manipulator. 

21.-25. Install support cables B, C, D, E, F. 

26. Inspect entire structure. 

In these examples no mention has been made of the manipulator control 
mechanisms, nor the method of sensing the various conditions upon which 
decisions are made or control is achieved. The idea was to describe these 
tasks without regard to the method of solution. For example, the "precise 
positioning" called for in certain assembly steps might in some cases be 
achieved through the use of sensory information , while for others a priori 
information will be adequate. Of course we have restricted ourselves to 
the Space Shuttle manipulation scenario in which manipulators are used 
to produce all motions and extra vehicular activity is not considered. 

The entire operation theoretically could have happened under contJrol 
of a teleoperator using only the operator's direct sensing and perceptual 
abilities. We are interested in discussing another alternative, where the 
operator takes little or no part in the activity once the decision is made 
to start a particular sequence. In this case computational machinery is 
responsible for determining the proper next step in a sequence, perceiving 
the state of the environment through sensors, controlling manipulator 
motions, and measuring the successful execution of each operation. 

At the present time there are no detailed plans for many of the 
manipulation applications to which the Shuttle may be applied. The 
mechanical designs for grapple fixtures, beam connection fittings, and 
even the gross features of structures await further development and plan- 
ning. This lack of specification has both positive and negative effects 
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on the present task of determining technology requirements for automated 
manipulation. Our efforts suffer from the difficulty of not knowing the 
precise nature of the assembly design; What motions are required? What 
will be the tolerances? Double insertions? Gripper requirements? At 
the same time we have the opportunity to influence the ultimate designs 
in each of these areas (connectors, motions, tolerances), in such a way 
that our job will be made easier. The current underspecification opens 
the way to considering assembly problems and the results of laboratory 
experience before the mechanical designs are complete. 

GENERIC ASSEMBLY OPERATIONS 

Using the example tasks as a guide, we can divide the components 
of the assembly process into five generic operations; transfer, grasping, 
mating, fastening, and inspection; 

Transfer 


During transfer the manipulator hand and its contents are transported 
from one location in the assembly area to another. In many cases the start- 
ing or ending point for a transfer raction are run-time variables since 
they may be specified from sensory information rather than from advanced 
planning knowledge. When manipulated objects are in motion, initial or 
final manipulator velocities may also be run-time variables. Therefore, 
initial and final "state" of the manipulator are the basic parameters 
of a transfer motion. 

Sinr'^ in>nsfer is characterized by large manipulator motions, an 
Important requirement is that no unexpected collisions take place among 
the manipulators, the manipulated objects, and other parts of the Shuttle. 
Overall speed and smoothness may also be important, as well as other 
trajectory attributes. 

For examples of transfer see task steps A-2, A-5, B-1, B-5, C-5, C-9. 


Grasping 


Grasping is the operation of temporarily attaching a manipulated 
object bo the end of the manipulator. Before grasping can occur, the 
manipulator hand must be placed in precise position relative to the object. 

If the object is moving the hand must also move to maintain the correct 
juxtaposition of surfaces. Then a mechanism is activated which performs 
the actual attachment. 

Sensors that help to achieve the required relative positioning 
are important to grasping, as is consideration of suitable grasping sur- 
faces on the manipulated object. Availability of good grasping surfaces 
can relax requirements on positional accuracy and provide a more rigid 
attachment. Grapple fixtures, mechanical devices attached to the manipulated 
object, are designed to provide ideal grasping surfaces, often yielding 
relaxed positioning requirements, rigid attachment, and facilitated 
sensory processing. 

For examples of grasping see task steps A-3, A-6, B-2, C-2, C-12. 
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Mating 


Perhaps the most basic step in mechanical assembly is the mating 
operation.. By executing "controlled collisions”, points, edges, and 
surfaces on two or more objects are brought into contact in a prescribed 
manner. Within the category of mating, insertion forms the largest and 
most typical subclass. Mating operations generaily found in the factory 
require 0.1 to 1.0 rail accuracy. Since this is, well below normal human 
visual resolution and the hands obscure vision during manipulation, 
force sensing plays a crucial role. 

For examples of mating see task steps B-5, C-3, C-10. 

Fastening 

Fastening is the process of making a mate permanent. The basic 
goal is to maintain permanent contact between two or more objects. Typical 
of the fastening operation is the fastener — an additional part or parts 
that are introduced to make the fastening possible. Transfer, grasping, 
mating, and inspection often play major roles in fastening. 

For examples of fastening see task steps B-6, C-6, C-10, C-21. 
Inspection 

By obtaining sensory Information from the task environment, 
the success of a previous operation is tested. Optical, mechanical, 
and electrical sensors may be employed. However, inspection is often 
more than just a passive sensory process — the response to a stimulus 
is often required (e.g. checking the tightness of a cable). To obtain 
robust and reliable performance, inspections must be performed continu- 
ously throughout an assembly. ^ 

If the assembly operations required for the three examples, 
tasks A, B, and C, are examined in terras of required resources (types 
and numbers of sensors, amount of sensory processing, precision of move- 
ments required , and special fixturing) , we can summarize our findings, 
as shown in Table 1. Each of the situations depicted on the left exempli- 
fies a handling task that needs lower accuracy, fewer sensors, less 
control computation, or reduced demand on special purpose hardware. 

Tasks involving those on the right place highe” demands on the system. 

Table 1, Task variables that affect requirements 


grasp using fixture 

<==> 

free grasp 

compact object 

<==> 

elongated object 

low mass object 

<==> 

heavy object 

object stationary during grasp 

<==> 

object moving 

single point mating contact 

<==> 

multipoint mating 

parts attached to Shuttle 

<==> 

floating parts 

slow movement 

<==> 

fast movement 

activity inside bay 

<==> 

outside bay 

no fixturing 

<==> 

fixturing required 

one manipulator 

<==> 

multiple manipulators 
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TASK PROBLEMS PECULIAR TO NASA 

Most of the teohnoloRy and conceptual approach described in 
this article closely parallel attempts to automate tnechanical assembly 
and handling operations in Earth-bound industry. In taking this approach 
we benetit from extensive experience derived in dozens of laboratories 
and factories over the last decade. In extending this technology to 
space, however, we encounter new problems for which no experience now 
exists. 


First consider the large dimensions. Virtually every successful 
use of manipulators on Earth has involved handling of rather compact ob- 
jects in a restricted region of space (i.e. the work station) [8, 1^,21, 
25,31(35,^9,51]. No large moments are generated and precision is only 
required near the hand. Note the following difficulties that arise 
when handling long beams of the type found in every plan for large space 
structures ; 

• Small errors in angular positioning of the manipulator wrist 
result in very large errors in the endpoint positions of 

the workpiece. These endpoints arc the very spots where part 
mating and connection must take place. 

• Solid gripping is made more difficult by large moments in 
the gripping plane. Resulting slippage is magnified at 
the beam endpoints. 

• Beams often require precision placement of both ends simulta- 
neously [6], A single visual sensing system oannot, in 
general, monitor activity at both ends of the beam at the 
same time . 

• Two-handed manipulation may minimize certain of thebe problems, 
but this is essentially a new technology [ 26 ], 

Though certain advantages are expected from manipulating in 
a zero gravity environment (0-g), (ie. primarily an increased load 
carrying capacity), there will also be complications. Since nothing 
can be left "sitting on the workbench”, large demands will be placed 
on fixturing, parts holders, and other restraint mechanisms. The mani- 
pulation system will be responsible for maintaining contact between 
mated parts until a fastening procedure is performed, since there are 
no natural forces performing this function. Perhaps the most troublesome 
fact is that pccurate simulation of a 0-g environment is not possible 
on Earth. Thu means that the first few assembly experiments actually 
conducted in space will be very risky learning ventures. 

Achieving economy of energy consumption during a manipulation 
procedure is another factor of little concern in the factory or laboratory, 
but of substantial importance in space. Task strategies and manipulator 
trajectories should be selected to minimize energy usage. Though a 
wealth of literature exists in the general area of optimal control, 
few attempts have been made to apply these techniques to multi- joint, 
serial link manipulator control [23, 43], 
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III. REQUIRED TECHNOLOGY 

To understand how these tasks can be accomplished in the manner 
indicated, it will be useful to describe briefly the major components 
of an automated assembly system, and the technological constituents 
of each. The automated assembly system is composed of four major elements; 

A supervisory element, an element that perceives the state of the environ- 
ment, an element that affects the environment, and the assembly task 
Itself (see Fig. 4). 

Research in machine intelligence and robotics has contributed 
extensively to our understanding of these four areas during the past 
decade. As a result many of the techniques needed for Shuttle automation 
are already available in the laboratory and some have been accepted 
in the factory; they need only be refined and adapted, to the special 
rigors of space usage to be useful. Other areas are less well developed 
and will require additional basic research, experimentation, and new 
technology, in addition to space qualification. The Appendix gives 
a brief assessment of the state of the art. 

In the following sections I will deu'cribe how technical developments 
from each of these four areas contribute to automation of mechanical 
assembly procedures. I concentrate on specific questions for automation 
of the Space Shuttle manipulation system: What technology is needed? 

What technology is well developed? What looks ripe for development? 

What problems have no solution in sight? Here again, specific application 
tasks are quite important and the three example scenarios are referenced. 

AUTOMATED SUPERVISOR 

When a teleoperator performs an assembly operation the manipulator 
executes a sequence of actions that result in achievement of some goal. 
During execution, the human specifies which action is taken next in 
the sequence, his decision often depending on the outcome of a previous 
step. The division of function between the manipulator and the human 
is often called supervisory control. For a robot, a similar supervisory 
function is needed, but it is no longer performed by a human. Instead, 
an automated supervisory computer program invokes the sequence of actions, 
evaluates sensory information, makes its own/'decisions, and inspects 
its own work [36, ^0, 1?]. 

One important function of this supervisory software is to permit 
efficient communication between humans and the automated system. Telling 
a robot what one wants it to do can be a difficult job. We liken the 
human telling the supervisor what the robot should do, to a plant manager 
telling a foreman what his group is to do. A special language is required 
that incorporates a vocabulary rich enough to describe what needs to 
be done, but with as little conversation as is reasonable. Clearly, 
a more responsive and intelligent foreman makes the manager's job easier 
and more pleasant . 

This process of interaotion between the human and assembly 
system will take two forms. During mission planning, technicians will 
teach the system how to perform a particular assembly. First the logical 
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Figure Major Components of an Automated Assembly System 
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sequence of actions, (grasps, transfers, insertions, inspections, etc.) 
are specified in symbolic form. Then data are trained that specify the 
precise details of part and fixture locations, and important dimensions. 
Many powerful techniques already exist for this type of instruction, 
though additional consideration may be required for the Shuttle applica- 
tion, since training will take place in 1-g and actual performance 
takes place in 0-g, 

The other form of interaction takes place during a mission. 

Here the assembly supervisor keeps the operator informed of the system's 
operational status, especially when unusual events occur. It also 
provides a facility that permits the operator to intervene with changes 
to the original assembly plan, or modifications to a plan gone wrong. 

This option of interacting with the behaving assembly system is very 
important. 

The other important function of the supervisor is to provide 
coordination among system components. Since an assembly system is • 
typically composed of a number of subsystems, both hardware and software, 
achieving any result requires careful orchestration. Even telling 
such a system to STOP will often require issuing a detailed set of 
instructions. 

An assembly supervisor for the Space Shuttle should perform 
the following combination of functions; 

• Detailed Information about the status of the system is made 
readily available in an easily understood form. 

• Self documenting, easy to generate commands can be given 
in a teaching mode and during run time. 

• Changes can be made to previously determined plans. 

Strong support for interaction between the operator and the 
automated supervisor are required to obtain the best performance from 
both the man and machine. Current teleoperator techniques (especially 
raanipulanda) do not permit intimate human-computer interaction, because 
the operator is busy with special purpose joysticks, knobs, and buttons. 

A keyboard permits a more general class of commands, though analog 
instruction is difficult. 

Special safety features will be required for a Shuttle supervisory 
system. These safety features are generally unnecessary and often 
undesirable in the laboratory, and will, therefore, require special 
attention. 

When confronted with unfamiliar assembly tasks, humans are reliably 
able to generate successful assembly plans that specify ordered sequences 
of operations, effective inspection procedures, and error recovery 
strategies. In this section we have described an automated assembly 
supervisor that includes 'none of these abilities. All planning, sequence 
generation, and error recovery techniques will be worked out in advance 
and made available to the robotic system by humans. Though automated 


planning systems have received substantial research attention in the 
past ten years, [15, 16, 30, 31, ^2, i<9], the remaining problems are 
difficult and the field is still rather immature. Other problem areas 
may find suitable applications for such planning systems before 1990, 
but automated mechanical assembly in space will not, 

MANIPULATOR CONTROL 

In order to allow a manipulator to operate independently of 
direct human control, it is necessary for motions of the manipulator 
to be controlled by a computing system. Such a system must have a 
trajectory planner that satisfies the goals of each movement without 
undesired collisions and a control program that moves the manipulators 
along planned trajectories. Trajectories must ensure collision- free 
transit, and energy minimization may be desirable. The controller 
must compensate for certain of the dynamic properties of the manipulator 
including those changed by grasped objects. 

Automatic trajectory control is probably the single most useful, 
best understood, and most easily implemented technique that can be applied 
to the Shuttle assembly system, A wide range of functions can be performed 
using a trajectory controller; a human operator can invoke an automatic 
movement to a much used position. The operator can routinely specify 
movements in terras of desired position, letting the system determine 
and execute the movements. Or the system can have a sequence of operations 
to perform that invoke movements through such a trajectory control 
system. The entire range of autonomy, from advanced teleoperator, 
the full fledged robot will make use of the trajectory controller. 
Therefore its development is central to any automation plan. 

A benefit of automatic trajectory generation is the opportunity 
to use energy-efficient movements. Because the Shuttle manipulator 
is so large, and payloads may approach 60,000 Ibs-mass, large energy 
savings are possible if operators do not "jocky" the payloads about. 

An automated trajectory planner can use standard techniques to generate 
energy minimizing manipulator trajectories. 

Obtaining solutions to minimization criteria during an assembly 
may not be feasible due to computational limitations, but precalculated 
results could be useful for certain sets of movements. There may also 
be general "modes" of operation or regimes that yield lower cost motions 
on the average. In any mode where joystick commands./do not directly 
determine the exact path of manipulator movement there is room for 
automatic reductions in energy usage. Permitting specification of 
via points and goal end-points would allow incorporation of the human’s 
ability to choose obstacle- free paths, while including some degree 
of energy optimization. 

Most manipulators used for Earth applications employ simple 
servos closed separately about each manipulator joint [24]. Changing 
moments of inertia, inertial coupling between joints, velocity interaction 
terms, and linkage flexibility are ignored by these controllers, while 
feedback is used to eliminate the adverse affects of all factors. 

The resulting level of performance is adequate for many applications. 



especially where arm velocities are low enough so that centrifugal 
and Coriolis forces are of little consequence. In other cases accuracy 
requirements are sufficiently low so that more sophisticated control 
is not needed. Under other circumstances, however, simple servo controllers 
yield sluggish and insufficiently precise performance. It is notable 
that at least one large reach, large mass Industrial manipulator must 
be operated below maximum velocity (determined by actuator strength) 
in order to meet stability requirements. 

For control of the Shuttle manipulator there are a number of 
factors recommending compensation for some of these dynamic terms. 

Its size is one obvious factor. Large moment arms produce strong 
inertial coupling forces that perturb operation. In many instances 
the precision required during grasping or mating operations will be 
high enough to make compensation essential. Finally, plans to handle 
very large payloads will require a more comprehensive compensation 
algorithm, since critically damped loops may depart markedly from desired 
performance when the mass of the hand changes by several orders of 
magnitude. 

This last problem has strong implications for the teleoperator. 

Due to the long time constants associated with large masses, predictive 
corrections will be required over intervals of many tens of seconds. 

Though training can compensate for the difficulty of performing in 
this circumstance, safety considerations will demand very slow operation. 
Using automatic techniques, the throughput of the system can be greatly 
enhanced while maintaining adequate safety margins. 

When a manipulator is moved about it is desirable to prevent 
it from unexpectedly colliding with other objects in the environment. 

There are two general schemes for protecting against such inadvertent 
incidents. The most general approach is for a geometric model of the 
manipulator's environment to be available to the trajectory planning 
system [30]. In this model is represented each potential obstacle 
and the manipulator itself. In order to choose a collision free path 
a computing element checks for intersection of the manipulator body 
with the space occupied by each of the potential obstacles. Though 
a number of heuristics exist for simplifying this procedure, it usually 
requires an unacceptable amount of computer time and memory to be employed 
in its most general form. 

Another solution is to plan the environment so that safe passage 
is ensured if certain rules are followed . Restricting all objects to 
rest on the work surface, and causing all trajectories to move "up", "over", 
and "down" is such an approach. Intermediate between these approaches 
are those that only model each obstacle roughly, using some sort of con- 
venient envelope, and then use a restricted set of trajectories designed 
to avoid these envelopes [29]. 

It may also be- possible to devise a sensory system that can 
detect impending collisions early enough during a motion to avoid forceful 
impacts. Proximity sensors come to mind for such an approach. 


SENSING 


Sensors and sensory processing give an automated system information 
that can be used to compensate for uncertainties in the environment 
and uncertainties in the automated system itself. Perfectly precise 
manipulators acting in a perfectly ordered environment do not need 
sensors, especially if 100$ reliable operation is not required.- Indeed, 
a number of industrial robotic applications function quite respectably 
in a virtually open loop manner. Yet the cost of such operation, in 
terms of precision hardware, constraints on organization of the environ- 
ment, lack of flexibility, and the impact of failures, can be quite 
high. 

In terms of the Shuttle manipulation problem there are three 
basic reasons why sensing is required; 

Not every aspect of the environment will be under NASA's 
control during manipulation operations. The hcf-nario 
task of satellite recovery is a good example; here, 
the exact position and movement of the payload is not 
available from a priori knowledge. It must be obtained 
by examining the environment with sensors throughout 
the manipulation period. 

Manipulators and accessory equipment of sufficient precision 
to obviate the need for closed loop operation^ are too 
heavy, cumbersome, and lack the versatility needed to 
economically fly on a spacecraft. 

- ■•r'' 

Robust and reliable performance will be of paramount 
importance during Shuttle manipulation procedures. 

Only through effective sensing can the small variations 
that cause failures be detected and accommodated, and 
in this way only can the failures themselves be detected 
once they occur. 

It must be made clear that transduction and communication play 
only preliminary roles in effective sensing. Once sensory data are 
made available they must be processed to yield useful information, 
and then useful results. It is this computational factor, often ignored 
or slighted, that makes effective operation difficult to achieve. 

There are five different sensors falling into two categories 
that are most useful for robotics applications. The first category 
of sensor, "terminal sensors", are carried on the manipulator hand, 
or end effector. They provide data about the relationship between 
the environment and the business end of the arm. 



"iThe manipulator has internal position and velocity sensors that are 
always used in controlling movements. But control is not always 
closed through the environment. 
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Force sensing gives the manipulation system information about 
the mechanical coupling that occurs when the hand comes into contact 
with objects. Typical force sensors incorporate strain gauges mounted 
on the fingers of a gripping device, or in the arm’s wrist. They indirectly 
monitor forces and torques by measuring deformation of known mechanical 
elements. The resulting information can be used directly in the feedback 
loop of a manipulator control algorithm [26, 32, 38 ? 50], or in generating 
knowledge about the manipulative state [25, ^4], 

Touch sensors are special case force sensors that indicate 
contact of the hand with the work environment. Additional information 
about the location and pattern of contact is also often provided [20]. 

These discrete data can be used to detect obstacles, verify the success 
of a grasping action, or detect slippage between the hand and a grasped 
object. 

Proximity sensors tell the manipulation system when objects 
are in the vicinity of the sensor. These devices can be used to avoid 
obstacles and to precisely locate objects for grasping, or other forms 
of handling [5]. A prototype set of proximity sensors have been used 
in a teleoperator mode with the JSC training arm. Preliminary reports 
show them to be quit's effective in aiding a passively cooperative grappling 
operation [7] . 

The second category of sensing system is based on optical imaging, 
and can be loosely called computer vision. Here the overall environment 
of the manipulation system can be monitored for a variety of factors. 

As with the force sensor, information from a computer vision system 
can be used directly in low level control algorithms [9, 19, i<0], or 
in acquiring knowledge about the task environment so that decisions 
can be made [22]. 

Many different basic approaches exist for computer vision, 
encompassing a variety of hardware configurations and algorithmic techniques 
[11, 52]. For the Shuttle we emphasize only two sets of techniques: 

• raster scan video devices that generate two dimensional 
digitized grey-level images at normal video rates, and 

• laser ranging devices that generate depth maps of objects 
in the environment. 

Once again 1 emphasize that Imaging, transduction, and communica- 
tion of image data comprise only minor components of the computer vision 
challenge. Powerful computational elements are required to make useful 
perceptions upon image data in an acceptable amount of time. In most 
cases there are only two alternative means of providing useful real-time 
service from a vision system: 1) use extremely simple algorithms that 

execute with sufficient speed on existing general purpose computers, 

2) implement more demanding algorithms with special purpose high speed 
computational hardware . 

Solutions to the general vision problem, i.e., where objects, 
backgrounds, lighting conditions, and object motions are unconstrained, 
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do not now exist (except in biological form). By labeling objects, 
constraining the background, manipulating lighting conditions, and using 
a priori knowledge about object motion we obtain a tool of unparalleled 
usefulness. Indeed, the fact that the user of a teleoperator relies 
heavily upon his visual sense is a good indication of its importance 
to effective manipulation. 

In addition to using vision for direct servolng of the manipulators 
and for allowing freedom in the placement and orientation of parts, vision 
can be used in situations where a priori knowledge of a task parameter is 
not available. Position and motion of the satellite in Task A, discussed 
earlier, is an example. Vision can also contribute to the Shuttle’s ability 
to maintain position and orientation relative to other spacecraft, platforms, 
or structures. When manipulating with respect to such structures, residual 
errors in Shuttle stability can be detected and their effects nulled by 
modifying manipulator control signals. 

The potential for computer vision in automated assembly and 
other application areas in space is enormous and exciting, but should 
not be cataloged here. For more extensive, but general accounts see 
Winston’s text [52], 


MECHANICAL CONSIDERATIONS 

A long term goal of robotics research has been to develop machines 
that have the flexibility to perform all the functions required in 
a particular domain, without resorting to the use of too many special 
tools, fixtures, or other paraphernelia. While the costs of fixed 
automation on Earth can only be justified for very high volume production, 
the cost of versatile machines can be amortized over production of 
a wide variety of low volume products. Similar arguments apply to 
automated mechanical assembly operations performed in space. Versatility 
is desirable for apace applications since a wide variety of relatively 
low volume tasks are to be performed, and the costs of putting additional 
hardware into orbit are high. 

Three questions are of importance to these versatility Issues; 

1) Where can the manipulators reach and with what orientations? 2) 

What set of grippers and special tools are needed to perform all necessary 
functions? 3) How much special purpose fixturing and jigging is required? 
Naturally, little can be decided without detailed information about 
the task set, but we can make some general preliminary observations. 

It is suspected that for all but the most trivial tasks two 
manipulators are an absolute minimum. Working through example Task 
B one sees that a substantial amount of fixturing is required to holi 
subassemblies in position while additional parts are added. Some of 
these fixtures could be replaced with additional, smaller manipulators 
located in the bay area. If large packages can be precisely positioned 
only when handled by two arms, additional devices will be required 
to perform fastening operations. 
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Though conventional wisdom recommends the use of manipulators 
having six degrees of freedom, this should bo considered a lower li« . 

Six Jointed arms permit the hand to be positioned and oriented arbit. -rily 
within the work space. ^ However, the space occupied by the body of the 
manipulator is not so unconstrained, Only manipulators having a so- 
called redundant design, i.e., more than six joints and links, can reach 
points in space with a range of configurations (e.g., freedom to posi- 
tion the elbow away from obstacles). When the manipulators are used to 
reach points within a frame-like structure , or any cluttered environ- 
ment, this ability to select from sets of arm configurations will be 
quite valuable. These arguments are of little immediate relevance here 
since the manipulators currently planned for the Shuttle are not redun- 
dant designs [13]* 

An ideal manipulator system would use a single very dexterous, 
very sensitive hand to perform all manipulative tasks. However, end 
effectors of sufficiently sophisticated design do not currently exist; 
most manipulators use mechanisms having Just two or three very simple 
"fingers”, each with a single articulation. 

One way to compromise the lack of versatility demonstrated by 
today's grippers with the demanding characteristics and requirements of 
an assembly task is to use sets of interchangeable, special purpose end 
effectors. This technique permits each end effector to be of a very 
special-purpose design while the overall assembly system retains its 
versatility. This concession to specially designed hands is not, however, 
made easily; 1) A means of changing end effectors is required ~ a 
requiremefit complicated by the presence of sensors and their electrical 
and possibly optical connections to computational equipment; and, 2) a 
great deal of time may be required for changing grippers [34]. Unfortu- 
nately grippers, sensors, and control mechanisms sufficiently versatile 
to obviate the need for this strategy do not yet exist. 

As advanced gripper designs include more and more sensing elements, 
the problems of signal communication increase. Most present manipulators 
use a separate set of wires for each hand mounted sensory system. 

This results in difficult reliability problems when the wires are routed 
to processing electronics through some five or six mechanical articulations. 
Some sort of unified sensory communications system should be developed, 
possibly making use of custom designed integrated circuits and a single 
high bandwidth channel. 

Almost every point and issue in this section on mechanical con- 
siderations applies equally well to the teleoperator concept as to the 
automated system. Workspace analysis, gripper design, and fixturlng 
are task related problems that are indifferent to the implementation 
media of the controller — silicon or protoplasm. 


^This claim of generality for the 6 degree-of- freedom manipulator 
suffers from circularity, since the work space is usually defined 
in terms of the kinematic properties of the manipulator. 
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COMPUTATION 


From the foregoing discussion it should be clear that ccmputation 
plays a crucial role in almost every aspect of an autonomous assembly 
system, Trajectory control, obstacle avoidance , force sensing, energy 
minimization, task planning, sequencing, and computer vision are problems 
that might have trivial solutions were it not for unusually demanding 
computational requirements. Any attempt to automate the Space Shuttle's 
assembly system must allow for a sizeable investment in machinery that 
can satisfy these requirements. 

Recent advaiiOes in computer electronics have produced a host of 
new components (processors, memories, logic elements, etc.), that are 
faster, more powerful, consume less energy, cost less, and are lighter 
than ever before. The use of custom designed large scale integrated 
circuits (LSI) can extend these economies even further. With these 
advances the problem of providing the Shuttle with adequate computational 
facilities for autonomous operation is more one of algorithm and software 
development than of electrical or computer engineering, though certain 
system integration issues are still important. 

The computers required to perform computational functions for 
sensing, control, and supervision fall into two basic categories; general 
purpose stored program machines and special purpose hard-wired processors. 
General purpose machines are needed for functions that depend on a variety 
of sequential operations, especially when branching is common. They can 
be used when real-time demands are only moderate, or when aided by other 
computational devices. Special purpose processors can be very fast, so 
they are used where algorithms are time-critical. The speed of the 
special processor is obtained at the expense of flexibility ~ they are 
generally restricted to implementation of algorithms that are very regular 
and do not depend on a large variety or large number of sequential steps. 

Though no precise prediction can be made here, the following 
is a brief account of computational requirements for automation of 
the Shuttle: 

• An automated supervisor will require a moderately large 
general purpose machine to support sequencing, safety 
monitoring, and interaction with the user. 

• Manipulator control can best be achieved by combining 

a general purpose computer, used for trajectory generation, 
energy minimization and obstacle avoidance, with special 
hardware that actually controls the manipulator. 

• Computer vision in particular and sensing in general 

has requirements that also favor combined use of a general 
purpose computer along with dedicated equipment. A 
sequential machine is needed to perform recognition 
functions at an intermediate level on data that are 
pre-processed by a set of special purpose processors, 
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IV. 


SUhWARY 


The main thesis of this article is that the Shuttle manipulators 
can and should be automated so that humans need only play a guiding role in 
handling payloads and assembling structures in space, The four cornerstones 
of an automated mechanical assembly system were described — automated 
supervision, manipulator control, sensing, and the mechanical characteristics 
of manipulator and the task — with an introductory analysis of important 
development areas. These development areas include? 

• Highly Interactive assembly supervisors that coordinate 
subprocesses, permit the user to easily instruct the 
system, and allow control with safety during run-time. 

• High precision manipulator controllers that conserve 
energy and avoid obstacles. 

• Use of sensory information in situations that lack a 
priori description, for control during mating and fastening, 
and for inspection. 

• Redundant manipulator designs, dexterous grippers, and 
communication with hand mounted sensors. 
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APPENDIX 


STATE OF THE AUTOMATED ASSEMBLY ART 

Each of the four major technology areas of automated mechanical 
assembly and some of the necessary ccmiponents have been discussed in 
the text. We are left^-'^th the question, "Which of these components 
are available, and whi(_/ require more research and development to be 
useful?" None of these technologies has been flight qualified, though 
aChumber have found their way into industrial applications, j , 

In this Appendix , each technological component is listed with a 
code chat roughly indicates its state of advancement: 

Factory The technique is being used successfully in the 

factory, '' 

Lab Successful use in the laboratory. 

Exprerimental The problem is receiving experimental attention in 
the laboratory, but no working prototypes yet exist. 

0 An unexplored problem. 

- The problem lacks signs of near-term solution. 

? No data. 


AUTOMATED ASSEMBLY SUPERVISION 

Methods for off-line training of logic data 
and trajectories 

Sensory based decisions easily specified 

. to supervisor. 

. ;^pwerful tools for debugging and modifying 
assembly plans. 

Coordination of hardware and software 
subprocesses. 

Methods for insuring safe operation. 

Automatic assembly plan generation. 


MANIPULATOR CONTROL 
Servos 

PID servos - Each joint is controlled 

separately. No dynamic compensation. 

Paul servo - Each joint is controlled 
separately. Corrections for gravity and 
changing moment of inertia are generated by 
pre-computation. (Requires trajectory 
planning . ) 
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Factory 

Lab 


Factory 

Lab, Experi- 
mental 
Lab 

Lab, Factory 
0 

Experimental , 


Complete real-time dynamic correction. 

Two handed cooperative manipulation. 

Trajectory planner 

Playback of trained motion. 

Joint space trajectory planner. 
Coordinated joint motions in Cartesian or 
Joint space. Smooth start and finish 
with via-points. Off-line computation. 
Real-time coordinated joint trajectories 
with position/velocity via points. 
Energy minimizing trajectories 


Obstacle avoidance 

Trajectories hand picked to avoid obstacles. 
Simple UP-OVER-DOWN strategy for avoidance. 
Crude models of obstacles permit prediction 
of collisions. Alternative trajectory is 
not found. 

Detailed object models predict collisions. 
Use models with real-time trajectories. 
Sensors detect impending collisions. 


Controller Architectures 

Analog servos. 

Digital servos 

Hybrid digital+analog controller. 
Microprocessor per joint controller. 
Microprocessor per function controller* 
Very special purpose hardware 


SENSING 

Force 

Forces at hand are determined from motor 
currents. 

Forces are detected in fingers. 

Three forces and three torques are 
measured at the wrist. 

Forces and tor.jues measured in work station. 


Touch 

Micro-switches in the fingers detect an 
object. 


Experimental 

Experimental 


Factory 

Factory 

Lab 


Experimental 

0 ,? 


Factory 

Lab 

Lab 


Experimental, 
0 , - 
0 


Factory 

Factory 

0 

Lab, + 

Experimental 

0 


Lab 

Lab 

Lab 

Lab 


Factory 
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Touch sensor array gives information about 
location and pattern of touch. 


Proximity 

Ultrasonic sensor detects object between 
fingers. (Not usable in space.) 
Optical sensor mounts on fingers. 


Computer vision 

- camera arrangements 

Single fixed camera 
Computer controlled pan and tilt 
Computer controlled iris and focus 
Stereo pair of cameras 
Orthogonal set of 2 or more cameras 


- raw data type 

Binary images 

Grey level images (16, 6i|, or 256 levels) 
Color data 

Tirae-of- flight laser range data 
Triangulation laser range data 

- object recognition and description 

Two-dimensional scenes 
Three-dimensional scenes 
Detect identity position, and orientation 
of known objects when presented in 
isolation against simple background 
with controlled lighting 
Precise], y determine positiofn and orientation 
of known object when presented against 
complicated, but known background, with 
some control over lighting 
Clusters of different objects 
Shadows (special cases) 

Track motion of part 

Locate special label in cluttered scene 
Track moving label 


- manipulator control 

Automatic calibration of relationship 
between manipulator and camera position 
LOOK-COMPUTE-MOVE strategy used for 2-D 
visually controlled manipulator 


Lab 


Lab 

Lab 


Factory 

Laboratory 

Experimental 

Lab 

a 


Factory 

Factory 

Experimental 

Lab 

Lab 


Factory 

Lab 

Factory 


Experimental 


Lab 

Lab 

Lab 

Experimental 
Lab , Experi- 
mental 


Lab 

Factory 
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LOOK-COMPUTE-MOVE strategy used for 3-D 
visually controlled manipulator 
Pseudo real-time (0.5 - 1.0 sec per cycle) 
2-D visually controlled manipulator 
Keal-time 3-D visual control of manipulator 
with special markings 


- computation 

Dedicated mini- or micro-computer 
Large general purpose computer 
Array processing + general purpose computer 
Hardware feature extraction at video rates 


MECHANICAL CONSIDERATIONS — GRIPPERS AND FIXTURES 
Grippers 


Two finger grippers 

Interchangeable, special purpose grippers 
Three finger gripper 
General purpose gripper 
Advanced automated special purpose tools. 
These "hand" held devices must incorporate 
sensors, computation, and actuators 

Fixturlng (A well understood technology) 
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Lab 

Lab 

Experimental 


Factory 

Lab 

Lab 

Lab 


Factory 

Lab 

Experimental 

0 


Factory 
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